Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HTML: adding spectroscopy section to rendering #339

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

HTML: adding spectroscopy section to rendering #339

wants to merge 2 commits into from

Conversation

bsipocz
Copy link
Member

@bsipocz bsipocz commented Sep 19, 2024

This should not be merged just yet, but serves as a placeholder and testing ground to see how it will look like.

The notebook runs and rendered but is not particularly pretty, and also should go through a review before publishing on the website.

@bsipocz bsipocz force-pushed the HTML_add_spectra_render branch from f16c1b9 to 82f4fbd Compare September 30, 2024 20:53
@bsipocz bsipocz force-pushed the HTML_add_spectra_render branch from 82f4fbd to aa0f785 Compare December 11, 2024 22:35
@bsipocz bsipocz force-pushed the HTML_add_spectra_render branch from aa0f785 to ba334c7 Compare December 17, 2024 21:31
@bsipocz
Copy link
Member Author

bsipocz commented Dec 17, 2024

@troyraen @jkrick - here is the rendered version of the spectroscopy notebooks. There is a timeout in the section, otherwise it's nice that we have a plot at the end, independently of any issues with the previous cells.

@troyraen
Copy link
Contributor

Looking at the rendered version here I see that it's the JWST function timing out on the source labeled "JADESGS-z7-01-QU". That function takes about 15 min. on Fornax with most of that coming from this source and the next one ("TestJWST"). I'm not sure if we want to increase the timeout limit (if that's even possible) or just pick different targets. Might be a question for @afaisst.

@bsipocz
Copy link
Member Author

bsipocz commented Dec 18, 2024 •
edited
Loading

First run of the notebook run into this TimeoutError with the MAST cell (copied the traceback here, so we have it easily accessed once the CI is restarted):

%%time
# Get Spectra for JWST
df_jwst = JWST_get_spec(
    sample_table,
    search_radius_arcsec=0.5,
    datadir="./data/",
    verbose=False,
    delete_downloaded_data=True
)
df_spec.append(df_jwst)

Searching and Downloading Spectra... 
Processing source NGC3049
Number of search results: 0
Source NGC3049 could not be found
Processing source NGC4670
Number of search results: 0
Source NGC4670 could not be found
Processing source Tol_89
Number of search results: 0
Source Tol_89 could not be found
Processing source Arp220
Number of search results: 0
Source Arp220 could not be found
Processing source COSMOS1
Number of search results: 0
Source COSMOS1 could not be found
Processing source COSMOS2
Number of search results: 2
Number of files to download: 12
Processing source COSMOS3
Number of search results: 0
Source COSMOS3 could not be found
Processing source JADESGS-z7-01-QU
Number of search results: 101

---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
File <timed exec>:2

File ~/project/spectroscopy/code_src/mast_functions.py:48, in JWST_get_spec(sample_table, search_radius_arcsec, datadir, verbose, delete_downloaded_data)
     46 # Get the spectra
     47 print("Searching and Downloading Spectra... ")
---> 48 df_jwst_all = JWST_get_spec_helper(
     49     sample_table, search_radius_arcsec, datadir, verbose, delete_downloaded_data)
     50 print("done")
     52 # Group

File ~/project/spectroscopy/code_src/mast_functions.py:116, in JWST_get_spec_helper(sample_table, search_radius_arcsec, datadir, verbose, delete_downloaded_data)
    113     continue
    115 # Retrieve spectra
--> 116 data_products_list = Observations.get_product_list(query_results)
    118 # Filter
    119 data_products_list_filter = Observations.filter_products(
    120     data_products_list, productType=["SCIENCE"], extension="fits",
    121     calib_level=[2, 3, 4],  # only calibrated data
    122     productSubGroupDescription=["X1D"],  # only 1D spectra
    123     dataRights=['PUBLIC'])  # only public data

File ~/project/.tox/py311-buildhtml/lib/python3.11/site-packages/astroquery/utils/class_or_instance.py:25, in class_or_instance.__get__.<locals>.f(*args, **kwds)
     23 def f(*args, **kwds):
     24     if obj is not None:
---> 25         return self.fn(obj, *args, **kwds)
     26     else:
     27         return self.fn(cls, *args, **kwds)

File ~/project/.tox/py311-buildhtml/lib/python3.11/site-packages/astroquery/utils/process_asyncs.py:26, in async_to_sync.<locals>.create_method.<locals>.newmethod(self, *args, **kwargs)
     22 @class_or_instance
     23 def newmethod(self, *args, **kwargs):
     24     verbose = kwargs.pop('verbose', False)
---> 26     response = getattr(self, async_method_name)(*args, **kwargs)
     27     if kwargs.get('get_query_payload') or kwargs.get('field_help'):
     28         return response

File ~/project/.tox/py311-buildhtml/lib/python3.11/site-packages/astroquery/utils/class_or_instance.py:25, in class_or_instance.__get__.<locals>.f(*args, **kwds)
     23 def f(*args, **kwds):
     24     if obj is not None:
---> 25         return self.fn(obj, *args, **kwds)
     26     else:
     27         return self.fn(cls, *args, **kwds)

File ~/project/.tox/py311-buildhtml/lib/python3.11/site-packages/astroquery/mast/observations.py:517, in ObservationsClass.get_product_list_async(self, observations)
    514 service = self._caom_products
    515 params = {'obsid': ','.join(observations)}
--> 517 return self._portal_api_connection.service_request_async(service, params)

File ~/project/.tox/py311-buildhtml/lib/python3.11/site-packages/astroquery/utils/class_or_instance.py:25, in class_or_instance.__get__.<locals>.f(*args, **kwds)
     23 def f(*args, **kwds):
     24     if obj is not None:
---> 25         return self.fn(obj, *args, **kwds)
     26     else:
     27         return self.fn(cls, *args, **kwds)

File ~/project/.tox/py311-buildhtml/lib/python3.11/site-packages/astroquery/mast/discovery_portal.py:363, in PortalAPI.service_request_async(self, service, params, pagesize, page, **kwargs)
    360     mashup_request[prop] = value
    362 req_string = _prepare_service_request_string(mashup_request)
--> 363 response = self._request("POST", self.MAST_REQUEST_URL, data=req_string, headers=headers,
    364                          retrieve_all=retrieve_all)
    366 return response

File ~/project/.tox/py311-buildhtml/lib/python3.11/site-packages/astroquery/mast/discovery_portal.py:187, in PortalAPI._request(self, method, url, params, data, headers, files, stream, auth, retrieve_all)
    182 response = super(PortalAPI, self)._request(method, url, params=params, data=data,
    183                                            headers=headers, files=files, cache=False,
    184                                            stream=stream, auth=auth)
    186 if (time.time() - start_time) >= self.TIMEOUT:
--> 187     raise TimeoutError("Timeout limit of {} exceeded.".format(self.TIMEOUT))
    189 # Raising error based on HTTP status if necessary
    190 response.raise_for_status()

TimeoutError: Timeout limit of 600 exceeded.

@jkrick
Copy link
Contributor

jkrick commented Dec 19, 2024

If possible, I would vote for keeping that JADES target since it is a good test of the JWST system. I just looked it up and it is a faint lyman break galaxy for which we couldn't get spectra without JWST, so the perfect use case for wanting to use MAST. MAST has been helpful so far, so maybe we could ask them about the timeout error if that hasn't already happened.

@jkrick jkrick mentioned this pull request Dec 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
HTML rendering use case: spectroscopy Spectroscopy use case
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@bsipocz @troyraen @jkrick